Skip to content

A Run's outcome says whether its Engine step started, and the run record carries it (#757's first part, record 0201) - #770

Merged
don-petry merged 8 commits into
mainfrom
harness/engine-started-757
Nov 12, 2026
Merged

don-petry merged 8 commits into
mainfrom
harness/engine-started-757

Conversation

@don-petry

@don-petry don-petry commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

User description

Refs #757 ("A Run that ended environment does not use up the item's run ceiling"): its first part, by the Factory Builder's ruling of 2026-11-12. It does not close the issue; the counting is the next pull request, rebuilt from #769 on this one.

This changes harness/ and .github/workflows/, a model Lens's inputs: the Maintainer approves it by hand as donpetry-bot, and the Factory never approves it. It is held factory:hands-off.

What changes

The run job Hands on its Engine step's outcome as the output engine, ${{ steps.engine.outcome }}. GitHub skips the step once a step before it failed, as the CLI install did in the runner DNS outage
apply (harness/apply_step.py) Writes outcome/1's engine_started: false where the output is skipped, true where it is success, failure or cancelled, left out otherwise
outcome/1, run-record An optional boolean engine_started
The records port Reads it with a failed Run's outcome into the run record, and keeps it with a kept outcome from the last two pass records (records.engines, beside records.begun)
The pass's entry Hands engines to the records port
tools/check_workflows.py A job may hand on a step's outcome, one of four words GitHub gives, beside a step's own output; its copy of run-jobs.yml carries the new output
What reads it Nothing yet

It fails closed: the field says false only where GitHub said the Engine step was skipped. A missing output, a job killed before it handed on its outputs, an unknown word, or an outcome from before this change leaves it out, and the next change will count such a Run as one that may have reached the model.

The record is docs/decisions/0201-…. 0201 is the Factory Builder's number, behind 0197 (#774), 0198 (#761), 0199 (#760) and 0200. Until those merge, the checks legs are red on "records are numbered from 0001 with no gap", and nothing else fails. The number is confirmed before the final push.

Checks

  • Boundary tests, written first and watched failing: tests/test_steps.py::test_apply_says_whether_the_engine_step_started (each word, the live workflow's output and NEEDS), tests/test_adapters.py::test_whether_the_engine_started_is_read_with_the_outcome_and_kept_by_its_run_and_attempt, the outcome's own read in test_a_runs_outcome_is_read_from_its_one_file_and_held_to_outcome_1, the entry in tests/test_controller.py, and the expression rule in tests/test_tools.py.
  • 22 new mutants and 6 retargeted, each killed. make checks is clean but for the provisional number. The desk's unfiltered run at a46e5c7, the record renumbered in a copy to clear the gap: 6,900 of 6,900 mutants killed. After main's merge, every mutant of controller/__main__.py and adapters/records.py was run again: 205 of 205 killed.
  • Not verified on GitHub: that a job's output of steps.engine.outcome reads skipped after an earlier step failed. GitHub's documentation gives steps.<id>.outcome as one of success, failure, cancelled or skipped. No trial has run, since a session may not dispatch a workflow. If it reads otherwise, the field is left out, and the Run counts as it does today.
  • About 27,600 changed characters.

Protected paths

.github/workflows/, harness/, adapters/, controller/, schemas/, tests/ and tools/ are protected or reported paths: pre-approved by the Maintainer 2026-11-08, ruled by the Factory Builder (its ruling of 2026-11-12 on #757). harness/ and the workflows are model Lens inputs, so the Maintainer approves this pull request by hand.

Reviews

Reader Head Findings Disposition
Claude security pass b02ece0 0 clean
Claude review b02ece0 2 minor, 3 info Fixed in a46e5c7: the entry's reading from records before the pass is now asserted (its mutant restored); a counter-example holds the run record's engine_started to a boolean; the outcome read's guard has a test and a mutant; engines() reads the last record alone, since known keeps a fault from it alone and an earlier word could never reach a run record. Rollback: an older Release reads a newer outcome as no outcome/1, named as a cost in record 0201, as began_at already is
Carbon (gemini-4-carbon-high) b02ece0 3 minor, four parts read Fixed in dad9732: "failure" and "success" each have a mutant of its own. The previous[-2:] bound is gone, since engines() reads the last record alone (a46e5c7). The apply job's NEEDS: ${{ toJSON(needs) }} is held by the new test's read of the live workflow, and by every apply test, since without it nothing is read
Muse (muse-wonder-full-1.0) a46e5c7 1 major The numbering gap: by design. The record's number is the Factory Builder's, now 0201, behind 0197 to 0200
Astra (gpt-6-astra) dad9732 1 major Fixed in ed1a1f6: a kept outcome is tested against what was said of another Run and of another attempt, with a mutant that takes any Engine word kept

🤖 Generated with Claude Code


CodeAnt-AI Description

Record whether a run reached the Engine

What Changed

  • Failed-run records now say whether the Engine step started: false when it was skipped, and true when it succeeded, failed, or was cancelled.
  • If the step’s status is missing or unrecognized, the record leaves it unset rather than claiming the Engine did not start.
  • The status is retained when a failed run’s outcome is carried into later records.

Impact

✅ Identify runs that never reached the model
✅ Preserve Engine-start status across records
✅ Avoid false “not started” results when status is unavailable

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

…ord carries it (#757's first part, record 0200)

The run job hands on its Engine step's outcome; apply writes engine_started into
outcome/1, false only where GitHub skipped the step, and leaves it out where the
job said nothing. The records port carries it into the run record and keeps it
with a kept outcome. Nothing reads it yet. The record number is provisional.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner November 12, 2026 01:39
@don-petry don-petry added the factory:hands-off A person's mark: the Factory writes nothing on this pull request (kernel/terms.py LABEL_HANDS_OFF) label Nov 12, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-12T01:39:53.447539Z
We couldn’t complete this review. Comment @codex review to try again.
b02ece0 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
Pre-merge checks: Passed 1 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Passed Passed checks: 1 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@codeant-ai

codeant-ai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

CodeAnt  CodeAnt Review Status

✅  Review completed

Status Commit Started (UTC) Finished (UTC)
✅  #967f6 Incremental review completed c4d910d Nov 12, 2026 · 23:12 23:12
✅  #7390f Incremental review completed c033733 Nov 12, 2026 · 06:16 06:17
✅  #7ac67 Incremental review completed bf042b1 Nov 12, 2026 · 04:07 04:08
✅  #2271f Reviewed your PR b02ece0 Nov 12, 2026 · 01:39 01:44

@codeant-ai

codeant-ai Bot commented Nov 12, 2026

Copy link
Copy Markdown

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@codeant-ai codeant-ai Bot added the size:L This PR changes 100-499 lines, ignoring generated files label Nov 12, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements tracking of whether a Run's Engine step started (Issue 757). It introduces an "engine_started" boolean field in the run records and outcomes, which is determined by parsing the outcome of the "engine" step from GitHub Actions (via the "NEEDS" environment variable). The changes span across adapters, controller, harness, schemas, workflow validation tools, and include a new decision document and comprehensive tests. Since there are no review comments, I have no feedback to provide.

@codeant-ai

codeant-ai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

CodeAnt  CodeAnt PR Risk

🟢  Low Risk

  • The PR appears safe to merge; no actionable issues were identified in the review.

Assessed commit: c4d910d246c8

…art of it is held by a test (#757's first part, record 0200)

The Claude review's findings: engines() read an earlier record whose word could never
reach a run record; the entry's reading from records dated before the pass, the run
record's boolean and the outcome read's guard were held by no test. Record 0200 names
the rollback cost and what is not verified on GitHub.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed against the latest diff

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread tests/test_controller.py Resolved
Comment thread tools/check_workflows.py Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 15 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread harness/apply_step.py Resolved
don-petry and others added 3 commits November 11, 2026 20:23
's first part, record 0200)

Carbon's finding on b02ece0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…757's first part, record 0200)

Astra's finding on dad9732.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t, record 0201)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry don-petry changed the title A Run's outcome says whether its Engine step started, and the run record carries it (#757's first part, record 0200) A Run's outcome says whether its Engine step started, and the run record carries it (#757's first part, record 0201) Nov 12, 2026
…t, record 0201)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	adapters/records.py
#	controller/__main__.py
#	tests/test_controller.py
#	tools/mutants.py
don-petry added a commit that referenced this pull request Nov 12, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gives the Engine's word on a failed Run (#757's first part, record 0201)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t, record 0201)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	adapters/records.py
@sonarqubecloud

Copy link
Copy Markdown

@don-petry
don-petry added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit ed140d1 Nov 12, 2026
32 checks passed
@don-petry
don-petry deleted the harness/engine-started-757 branch November 12, 2026 23:52
don-petry added a commit that referenced this pull request Nov 12, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Nov 12, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

factory:hands-off A person's mark: the Factory writes nothing on this pull request (kernel/terms.py LABEL_HANDS_OFF) size:L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants